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FOREWORD 


This  report  covers  the  actions  that  have  transpired  during  the  third 
year  of  Project  ABACUS,  the  Army’s  program  for  the  development  of  a 
prototype  Computerized  Training  System.  it  includes  a narrative 
summary,  key  documents,  and  amplifying  annexes. 

As  a historical  document,  it  will  be  utilized  in  preparation  of  the 
final  project  report,  it  Is  also  meant  to  provide  the  current  reader 
with  an  understanding  of  the  progress  to  date  of  Project  ABACUS,  its 
present  position,  and  what  actions  are  anticipated  to  be  completed  in 
the  near  future. 


ROBERT  G.  FOSTER 
LTC,  SigC 

Program  Director,  Project  ABACUS 
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I.  INTRODUCTION 

The  mission  of  Project  ABACUS  is  to  design,  develop,  test  and  evaluate 
a 128-terminal  Computerized  Training  System  (CTS),  utilizing  the  multi- 
minicomputer concept.  At  the  conclusion  of  three  years,  the  project 
is  progressing  satisfactori ly. 

I I . BACKGROUND 

The  events  leading  to  the  implementation  of  the  CTS  project  and  the 
project  progress  through  the  first  two  years  are  documented  in  CTS 
Report  TR-73-4  "One  Year  Status  Report,  Computerized  Training  System 
Project,  Project  ABACUS"  dated  1 August  1973  and  CTS  ReporJ  TR-74-4 
"Second  Year  Status  Report,  Computerized  Training  System  Project, 

Project  ABACUS"  dated  1 August  1974. 

The  Product  Manager  Charter  and  the  DA  Management  Plan  were  reviewed 
as  required  on  the  project's  anniversary  data  of  1 August  1974.  There 
were  no  changes  made  in  either  document. 

& 

The  annual  review  of  the  Memorandum  of  Understanding  (MOU)  between  the 
Product  Manager  (PM)  and  the  Commandant,  United  States  Army  Signal 
School  (USASIGS),  was  made  and  the  new  MOU  was  approved  and  signed. 

A copy  is  at  Annex  A.  Significant  changes  to  the  MOU  are  as  follows: 

a.  Designation  of  the  US  Army  Southeastern  Signal  School  (USASESS) 
to  the  US  Army  Signal  School  (USASIGS). 

b.  Designation  of  the  Computerized  Training  System  to  Project 
ABACUS. 

c.  The  Product  Manager,  CTS,  assumes  responsibility  for  funding 
and  programming  for  FY75  and  FY76  as  pertains  to  hardware,  software, 
contracting  change  orders,  systems  maintenance,  and  communications 

i nsta I I at  ion. 

The  Commandant,  USASIGS,  will  provide  FY75/76  O&MA  funds  for  normal 
operating  supplies,  computer  consumables,  and  magnetic  device  media. 

The  mission  and  functions  of  Project  ABACUS  remain  unchanged  as  de- 
scribed at  Annex  A. 

III.  PERSONNEL 

COL  G.B.  Howard  has  been  designated  Commander,  US  Army  Training 
Support  Activity  (TSA),  Fort  Eustis,  Virginia.  He  continues  as  Product 
Manager,  CTS  Project  A3ACUS.  LTC  Robert  G.  Foster  is  the  Program  Direc- 
tor and  Contracting  Officer  Technical  Representative  (COTR),  Project 
ABACUS,  Fort  Gordon,  Georgia.  Mr.  Donald  A.  Kimberlin  is  the  Chief, 
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Technical  Applications  Division  and  alternate  COTR  for  Project  ABACUS, 
Fort  Gordon,  Georgia.  The  Office  of  the  Product  Manager,  Computerized 
Training  Systems,  Project  ABACUS,  Fort  Monmouth,  New  Jersey,  and  the 
Training  Aids  Management  Agency,  Fort  Eustis,  Virginia,  merged  and 
consolidated  to  become  the  US  Army  Training  Support  Activity  (TSA)  on 
1 July  1975.  The  present  organizational  relationships  are  shown  in 
the  TSA  Organization  Chart  at  Annex  B. 

The  staffing  for  Project  ABACUS,  Fort  Gordon,  Georgia,  provides  for 
10  military  and  6 civilian  spaces  as  shown  at  Annex  C.  Note  that  3 
positions,  2 military  and  1 civilian,  reside  in  TDA  spaces  within  the 
Evaluation  ana  Studies  Office  and  Computerized  Training  Systems  Direc- 
torate, TSA,  Fort  Eustis,  Virginia,  with  duty  station.  Fort  Gordon, 
Georgia.  At  the  present  time  13  military  and  6 civilian  personnel  are 
assigned  to  the  project  with  duty  station  at  Fort  Gordon,  Georgia.  A 
roster  of  personnel  is  at  Annex  D. 

The  excess  military  are  programmers  assigned  to  the  project  to  keep 
programming  support  in  all  areas  on  schedule  with  the  time-phase  plan. 
The  area  of  systems  and  applications  programming  was  indicated  as  a 
potential  problem  area  in  the  Second  Year  S+atus  Report.  The  acquisi- 
tion cf  these  personnel  serves  to  alleviate  this  problem.  This  is  a 
one  time,  7 man  year  programming  effort. 

IV.  COMPUTER  SYSTEM 

Summary  of  procurement.  The  following  is  a summary  of  the  major  pro- 
curement actions  that  have  been  completed  prior  to  and  during  the  third 
year  of  the  project.  The  actions  described  at  subparagraphs  f,  g,  and 
h below  are  the  rasult  of  modification  changes  to  the  original  CTS  con- 
tract with  GTE-Sylvania  and  were  brought  about  by  the  Project  ABACUS 
move  to  Fort  Gordon,  Georgia. 

a.  April  1973  - Request  for  a Proposal  was  issued  to  industry. 

b.  December  1973  - GTE-Sylvania  was  awarded  the  contract. 

c.  April  1974  - Initial  computer  system  delivered  to  the  Office 
of  the  Product  Manager. 

d.  July  1974  - initial  32-terminal  display  controller-  delivered 
to  USASIGS. 

e.  May  1975  - Full,  six  processor  mu  I timin; computer  system  was 
delivered  and  inst.iled  including  16  terminals.  One  hundred  and  twelve 
terminals  were  placed  in  local  storage. 
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f . 

June 

1975 

g- 

July 

1975 

h. 

July 

1975 

- Communications  study  completed. 

- Communication  and  cabling  contract  negotiations. 

- FY76  Maintenance  of  CTS  contract  negotiations. 


Status 


I 


The  Phase  I Acceptance  Test  of  the  initial  32-terminal  system  was  run 
at  the  USASIGS  in  August  1974.  The  software  performed  to  standards 
and  was  accepted.  The  central  processor  hardware  did  not  perform  to 
standards  and  has  not  yet  been  accepted.  Also,  there  was  a manufac- 
turer keyboard  error  that  has  since  been  corrected.  The  Phase  II 
Acceptance  Test  of  the  system,  less  the  display  controller  at  the 
USASIGS,  was  run  at  the  GTE-Sylvania  facility  in  Needham  Heights,  MA, 
in  February  1975.  The  CLASS  I language  and  much  of  the  systems  soft- 
ware were  tested  at  this  time.  This  hardware  and  software  were  ac- 
cepted . 


The  project,  as  initially  planned  for  implementation  at  USASCS,  Fort 
Monmouth,  New  Jersey,  was  to  be  colocated  in  one  building  with  the 
three  selected  courses.  The  consolidation  of  the  Army  Signal  Schools 
at  Fort  Gordon,  Georgia,  has  resulted  in  the  computer  system  being 
located  in  a building  separate  from  two  of  the  courses  selected  at  the 
USASIGS,  Fort  Gordon,  Georgia.  The  computer  system  is  locatec  in  Moran 
Hall,  with  terminals  located  in  Moran  Hall,  Brant  Hall,  and  Greeiy  Hall. 
This  has  increased  terminal  communication  requirements  relative  to 
cabling  and  the  electromagnetic  environment  not  planned  in  the  original 
installation.  To  date,  a study  has  been  made  and  it  has  been  deter- 
mined that:  subterranean  cable  construction  is  required  between  build- 

ings since  there  is  no  existing  duct  work  available;  and,  amplifiers 
and  isolation  transformers  are  required  to  stabilize  the  signal  cir- 
cuits between  the  computer  and  the  remote  terminals.  The  results  and 
recommendations  of  the  study  have  served  as  a basis  for  the  following 
on-going  actions: 

a.  Modifying  of  the  original  GTE-Sylvania  contract. 

b.  Obtaining  the  necessary  funding  required. 

c.  Negotiating  the  contract  changes  with  GTE-Sylvania  and  US  Army 
Computer  Systems  Support  and  Evaluation  Agency  (CSSEA) . 

These  actions  will  enable  GTE-Sylvania  to  install  the  required  communi- 
cations for  the  CTS.  Additionally,  a contract  has  been  negotiated  for 
the  cabling  required  for  the  linkup  of  the  three  buildings,  and  the 
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cable  has  been  ordered.  The  inside  and  outside  cable  installation  is 
to  be  done  by  the  US  Army  Communications  Command  at  no  cost  to  the 
project . 

A fixed-fee  maintenance  contract  has  been  negotiated  to  provide  system 
maintenance  for  the  next  fiscal  year. 

The  Phase  II!  Acceptance  Test  has  been  rescheduled  to  take  place  after 
all  128  terminals  have  been  connected.  It  is  anticipated  that  this 
will  be  February  1976. 

To  date,  GTE-Sylvania  has  provided  all  the  training  required  by  the 
contract.  In  addition  to  this  training,  Project  ABACUS  personnel , 
under  a separate  contract  with  Digital  Equipment  Corporation,  received 
a special  two  week  programming  course  on  the  RSX-11D  operating  system 
used  with  the  CTS.  This  training  was  done  on  site  at  Fort  Gordon, 
Georgia. 

The  current  status  of  computer  system  contract  is  at  Annex  E. 

V.  COURSE  DEVELOPMENT 

Course  development  has  not  progressed  as  rapidly  as  expected  at  the 
time  of  the  publication  of  the  2nd  Year  Status  Report.  The  failure  to 
progress  as  anticipated  can  be  directly  attributed  to  a number  of  signi- 
ficant events  that  occurruu  in  the  second  year  but  did  not  impact  until 
the  third  year  of  the  project.  Major  among  these  events  was  the  move 
of  the  project  to  USASIGS,  Fort  Gordon,  Georgia.  Some  of  the  results 
of  this  move  were:  the  requirement  to  select  three  different  courses 

for  implementation;  establishment  of  a task  group  at  USASIGS;  movement 
and  recruitment  of  personnel  required  to  establish  a CTS  field  office; 
and  the  need  to  train  personnel  at  the  USASIGS  in  CTS  course  develop- 
ment during  the  critical  initial  stages  of  the  project. 

There  are  several  problems  that  have  contributed  to  course  development 
delays  that  are  endemic  in  an  operational,  military  school  environment. 
Among  these  problems  are  the  changes  in  course  programs  of  instruction 
because  of  changing  training  requirements  and  doctrine.  A case  in 
point  is  the  revision  of  the  Teletypewriter  Repair  Course,  MOS  31J20. 
After  approximately  a year  of  cevelopment,  the  POI  was  changed  to  ac- 
comodate new  equipment.  This  esulted  in  the  obsolescence  of  a signi- 
ficant quantity  of  prepared  lesson  material. 

Another  problem  that  is  common  in  a military  environment  is  the  change 
in  personnel  due  to  transfers,  reassignments,  and  expiration  of  terms 
of  service.  This  personnel  turbulance  necessitates  an  on-going  train- 
ing program  to  prepare  new  personnel  for  their  duties  with  a CTS  course 
development  project. 


A significant  amount  of  lesson  material  has  been  prepared  for  entry  into 
the  system  in  spite  of  the  problems  alluded  to  in  the  preceeding  para- 
graphs. However,  there  have  been  a number  of  factors  that  have  delayed 
the  entry  of  the  lesson  material  into  the  computer  system.  These  fac- 
tors were:  the  delay  in  site  preparation  for  the  full  128  terminal  sys- 

tem at  Fort  Gordon,  Georgia;  data  file  structure  of  the  initial  32  termi- 
nal system  installed  at  Fort  Gordon  was  not  compatible  with  the  file 
structure  of  the  final  system  and  a conversion  routine  had  not  been  pre- 
pared; poor  system  reliability  of  the  CTS  on-site  at  Fort  Gordon,  Georgia, 
after  the  move  from  the  contractor's  facility;  and  the  lack  of  entry 
specialists  for  on-line  entry  of  prepared  lesson  material. 

The  effects  of  the  above  cited  events  and  problems  have  had  a serious 
impact  on  the  original  milestones  in  the  time-phase  plan  for  course  de- 
velopment. This  has  necessitated  the  establishment  of  new  implementation 
dates  for  the  three  courses.  These  dates  are  as  follows: 


Course 


Original  Date 


Present  Date 


MOS  31E20 
MOS  31J20 
MOS  33L20 


August  1975 
October  1975 
November  1975 


April  1976 
June  1976 
July  1976 


The  presently  planned 
upon  the  system  being 
the  rescheduled  Phase 
plementation  schedule 


implementation  dates  were  established  predicated 
installed  in  the  fourth  quarter  of  1975.  However, 
ill  Acceptance  Test  date  may  cause  the  present  i in- 
to be  recons i dered . 


The  current  status  of  course  development  is  at  Annex  E. 


VI.  SYSTEMS  AND  APPLICATIONS  PROGRAMMING 


The  original  programming  responsibility  of  the  CTS  Field  Office  ^com- 
passed only  the  preparation  of  the  data  items  and  the  automated  data 
collection  routines  and  reports  required  to  support  the  project  evalua- 
tion. After  the  loss  of  key  personnel  in  the  Programming  and  Systems 
Operations  Division,  CTS  Product  Manager's  Office,  Fort  Monmouth,  NJ , it 
was  deemed  necessary  to  transfer  the  system  and  applications  programming 
effort  and  its  responsibilities  to  the  CTS  Field  Office. 

The  systems  and  applications  programming  effort  commenced  in  September 
1974,  with  the  assignment  of  the  ADPS  Operations  Officer  to  the  CTS 
Field  Office. 
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The  Project  ABACUS  programming  respons i b i I r ty  encompasses  the  following: 

a.  Prepare  the  applications  programs  required  to  interface  the  CTS 
with  the  on-going  train inq  program. 

b.  Prepare  the  macro  routines  reouired  to  implement  tne  instruc- 
tional decision  making  strati  :'^s  designed  with:n  the  instructional  model 
which  also  collect  real-time  student  aa+a.  These  data  are  used  to  gene- 
rate the  necessary  records  a-  J reports . 

c.  Generate  reports  requi"  : ttw  .cademic  records,  and  course 

development. 

d.  Prepare  the  automated  data  collection  routines  required  for 
course  development  and  resource  allocation. 

e.  Perform  the  s/stems  programming  required  to  modify,  update,  or 
generate  changes  to  the  CTS  operating  software  and  the  CLASS  I language 
systems . 

The  systems  and  app  I i ca+ : ons  procramminq  staff  was  comple+e  by  May  1975. 
However,  the  ADPS  Operations  Officer  is  scheduled  as  a loss  by  September 
1975.  His  replacement  has  been  requisitioned  and  will  be  aboard  by 
October  1975. 

The  work  is  progress i no  satisfactorily  in  thib  element  of  the  project. 

An  outline  of  the  currw  I t rt  is  Of  this  programming  effort  is  at  Annex  E 

VII.  PROJECT  EVALUATION 

The  Preliminary  Evaluation  Plan  has  been  updated  and  modified  to  an 
Operational  Test  Plan.  Due  to  its  preliminary  na* ure  and  the  number  of 
changes  incurred  by  The  project  since  the  initial  evaluation  plan  was 
prepared,  this  modi‘icaTion  was  considered  appropriate.  Some  of  the  pro- 
ject changes  which  imparted  on  the  evaluation  were  the  relocation  of  the 
project;  selectior  of  different  courses  of  instruction  with  an  attendant 
change  from  "lock-step"  courses  to  "self-paced"  courses;  decrease  in  anti 
cipated  use  of  tutorial  compu+er  applications;  and  a new  emphasis  on  Com- 
puter Manaqed  Instruction  (C'-'l).  Tne  main  effects  of  these  changes  have 
been  to  shift  the  evaluation  from  a rigorous  "lab"  test  of  CA I to  a pre- 
dominant operational  test  of  CMI.  It  is  anticipated  that  the  Operational 
Test  Plan  will  be  available  in  the  October  1975  time  frame. 

Despite  the  changes  tc  the  evaluation  plan,  certain  data  items  will  be 
requ i red  to  determine  attitudes,  course  development  time,  and  a number 
of  other  aspects  of  training  and  system  ( hardware/sof tware ) effectiveness 
These  data  items  have  been  stabi I i zed  based  on  known  requ i rements . The 
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surveys  for  students  and  instructors  in  the  on-going,  self-paced  cWrses 
and  the  CTS  instructional  programmers  have  been  completed  and  are  being 
administered.  Other  attitude  questionnaires  and  surveys  for  system  pro- 
grammers, console  operators,  school  staff  personnel,  and  course  manage- 
ment personnel  are  in  draft  stage  for  later  use  as  required.  Samples 
of  these  Questionnaires  are  at  Annex  F and  Annex  G. 


Student  base-line  data  has  been  collected  since  August  1974.  This  data 
will  be  used  to  assist  in  determining  the  training  effectiveness  of  CTS. 
Although  course  changes  have  taken  place,  a substantial  core  of  material 
will  be  available  for  a valid  comparison  of  training  effectiveness  be- 
tween the  current  self-paced  training  programs  and  the  CTS.  Data  col- 
lection will  continue  until  a sufficient  amount  is  accumulated  to  pro- 
vide a valid  sampling  base.  A number  of  aspects  of  the  training  effec- 
tiveness and  svstem  operational  test  have  been  expressed  in  the  form  of 
goals/objectives.  These  goals/objectives  and  their  collection  methodolo- 
gy have  been  incorporated  into  the  operational  test  plan.  The  training 
effectiveness  data  items  and  methodology  of  collection  are  at  Annex  H. 

The  cost  analysis  aspect  of  Project  ABACUS  is  being  conducted  in  three 
phases:  developmental,  capital,  and  operational  cost  phases.  Collection 

of  data  for  the  developmental  phase  is  complete  at  id  capital  costs  are 
currently  being  tabulated.  From  a larger  perspective,  the  cost  analysis 
will  encompass  descriptive  costs  ( budget/account i ng) , comparative  costs 
(effectiveness/benefits),  and  predictive  costs  (extrapol ation/forecast i ng) 
This  analysis  referred  to  as  the  Cost  and  Operational  Effectiveness  Anal- 
ysis (COEA)  represents  one  of  the  major  portions  of  the  operational  test 
plan  (i.e.,  the  "proving"  phase).  The  other  major  and  larger  section  is 
the  data  analysis  of  how  the  system  performed  including  problems  encoun- 
tered (i.e.,  the  "improving"  phase).  Both  sections  will  make  recommenda- 
tions for  the  future  of  Project  ABACUS. 


VIII.  CONCLUSION 

Taken  in  perspective.  Project  ABACUS  is  progressing  at  a satisfactory 
rate. 

There  have  been  a number  of  events  relative  to  relocation,  course  POI 
stability,  site  preparation,  system  failure,  terminal  communications,  et 
al,  that  have  been  addressed  and  their  impact  on  the  project  assessed. 
Each  event,  taken  individually,  would  not  have  been  a serious  obstacle, 
but  in  aggregate,  has  proved  to  be  too  formidable  to  overcome  within  the 
present  time  phasing  of  the  project.  No  extension  of  the  4-year  time 
schedule  has  been  requested.  However,  as  the  project  enters  the  fourth 
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year,  it  is  imperative  that  an  assessment  be  made  with  the  objective  of 
making  the  determination  of  the  impact  of  events  and  preparing  a realistic 
adjustment  of  milestones. 

As  the  project  progresses,  it  becomes  ever  more  apparent  that  fielding 
this  prototype  CTS  in  a dynamic,  operational  school  environment  will  pro- 
vide experience  and  lessons  in  dealing  with  problems  that  would  never 
arise  in  a research  milieu.  These  experiences  and  lessons  learned  will 
prove  invaluable  in  closing  the  research-user  gap. 


ANNEX  A 


MEMORANDUM  OF  UNDERSTANDING 
BETWEEN 


Product  Manager  Commandant 

Computerized  Training  System  US  Army  Signal  School 

Fort  Monmouth,  New  Jersey  Fort  Gordon,  Georgia 

1.  INTRODUCTION: 

a.  References: 

(1)  USASESS  Reg  10-2,  subject:  Organization,  Mission  and  Functions, 
dated  July  1973. 

(2)  Prototype  Computerized  Training  System  Management  Plan  (as 
revised  13  Nov  73). 

(3)  Product  Manager  Charter,  Prototype  Computerized  Training 
System  (as  revised  5 Nov  73). 

(4)  Letter,  ATSN-CTS,  USASCS,  29  Nov  73,  subject:  Designation 

of  Courses  for  Prototype  Computerized  Training  System  (CTS),  with  1st  Ind, 
ATTS-ITR,  TRADOC,  11  Dec  73. 

b.  In  accordance  with  cited  references,  this  Memorandum  of  Under- 
standing delineates  responsibilities,  command  and  control  channels,  and 
procedures  to  be  followed  in  the  operational  test  and  evaluation  of  a 
Prototype  Computerized  Training  System  (Project  ABACUS)  by  the  Office 
of  the  Product  Manager,  Fort  Monmouth,  and  the  US  Army  Signal  School 
(USASIGS),  Fort  Gordon,  Georgia. 

2.  GENERAL  RESPONSIBILITIES: 

a.  The  Commandant,  USASIGS,  reference  la  (1),  is  responsible  for 
preparing,  conducting  and  administering  course  of  instruction/programs 
of  instruction  (POI). 

b.  The  Commandant,  USASIGS,  reference  la  (2),  is  responsible  for 
monitoring  and  supporting  the  three  courses  reference  la  (4)  involved  in 
the  operational  phase  of  Project  ABACUS  for  the  Product  Manager. 


c.  The  Product  Manager,  CTS,  reference  la  (3),  is  responsible  for 
the  design,  hardware /software  development,  course  development,  opera- 
tion, and  evaluation  of  Project  ABACUS.  In  this  capacity  the  Product 
Manager,  CTS,  will  advise  the  Commandant,  USASIGS,  during  the  opera- 
tional test  period  in  those  areas  M responsibility  as  they  apply  to  the 
operational  test  and  evaluation. 

3.  DETAILED  RESPONSIBILITIES: 

a.  The  Product  Manager,  CTS,  will: 

(1)  P rocure,  deliver,  install  and  conduct  the  acceptance  test  of  the 
hardware /software  system  for  operation  at  USASIGS,  Fort  Gordon. 

(Z)  Provide  necessary  interfacing  between  USASIGS  and  other  activi- 
ties, elements,  and  contractors  in  matters  pertaining  to  the  prototype 
operational  test  and  evaluation. 

(3)  Develop  the  evaluation  plan;  designate  data  to  be  recorded  and 
required  format;  collect  and  analyze  test  data  and  prepare  the  formative, 
interim,  and  final  evaluation  reports. 

(4)  Prepare  appropriate  program  and  planning  documents  as  required 
by  and  for  submission  to  higher  headquarters.  Documents  applicable  to 

the  operational  test  and  evaluation  will  be  coordinated  with  the  Commandant, 
USASIGS.  The  final  document  shall  include  Commandant,  USASIGS 
concurrence/  comments . 

(5)  Establish  and  provide  to  the  Commandant,  USASIGS,  operational 
test  facilities  requirements. 

(6)  Provide  a training  program  for  USASIGS  system  and  course  person- 
nel in  the  system,  the  instructional  model,  and  course  development 
techniques . 

(7)  E stablish  and  maintain  a field  oLice  at  USASIGS,  Fort  Gordon, 
with  the  missions  and  functions  and  pha  f ed  personnel  augmentation  as  set 
forth  at  Inclosure  1.  The  organizational  relationships  between  the  field 
office  and  the  USASIGS  task  group  are  as  indicated  at  Inclosure  3. 


b.  The  Commandant,  USASIGS,  will: 


(1)  Provide  appropriate  facilities  for  the  operational  test  in  accord- 
ance with  requirements  established  by  the  Product  Manager. 

(2)  Coordinate  with  appropriate  Fort  Gordon  activities  to  support  the 
installation  of  the  hardware/software  system  at  Fort  Gordon,  as  required 
by  the  Product  Manager. 

(3)  Provide  administrative  support  for  the  acceptance  testing  at  Fort 
Gordon,  as  required  by  the  Product  Manager. 

(4)  Operate  and  maintain  the  hardware/ software  system  after  the  final 
contract  acceptance  of  the  system  by  the  Product  Manager  until  the  con- 
clusion of  the  operational  test  and  evaluation  with  the  monitorship  and 
advice  of  the  Product  Manager. 

(5 ) Plan,  program  and  budget  additional  system  capacity  not  required 
by  the  operational  test  and  evaluation.  Any  additional  use  of  the  system 

in  this  manner  will  be  limited  to  Project  ABACUS  applications  to  be  jointly 
concurred  in.  The  Product  Manager  CTS  will  retain  approval  authority  for 
recommended  system  use  that  is  in  addition  to  the  test  and  evaluation. 

(6)  Accept  full  responsibility  for  the  system  established  at  Fort  Gordon 
at  the  conclusion  ox  the  operational  test  and  evaluation  lor  use  in  accordance 
with  TRADOC  regulations  in  effect  at  that  time. 

(7)  Provide  the  USASIGS  Task  Group  leader  and  implementation  staff 
as  indicated  at  Inclosure  2 to  conduct  the  course  development  and  opera- 
tional test  of  the  prototype  system. 

(8)  Prepare,  conduct  and  administer  the  POI  for  the  operational  test 
and  evaluation  with  the  advice  and  monitorship  of  the  Product  Manager. 

(9)  Record  evaluation  data  under  specified  training /testing  conditions 
in  the  required  format  as  designated  by  the  Product  Manager. 

(10)  Provide  administrative  working  space  for  the  PMO  field  liaison 
office  and  coordinate  normal  post  support. 


4.  OTHER: 


a.  Funding:  Product  Manager,  CTS,  is  responsible  for  program- 
ming and  funding  for  FY  1975  and  1976  as  pertains  to:  hardware  and 
computer  software  acquisition  and  associated  contract  change  orde.  s, 
systems  maintenance,  interconnecting  video  and  signal  cables  between 
the  display  controller  and  student  terminals.  The  Commandant,  VJSASIGS, 
will  provide  FY  75/76  O&MA  funds  for  normal  operating  supplies,  com- 
puter consumables,  and  magnetic  devices  (i.e.,  magnetic  tape,  discs, 
and  disc  packs).  The  Commandant  will  also  provide  for  sy  item  operations 
and  maintenance  effective  1 July  1976. 

# 

b.  Responsibilities  for  items  not  specifically  covered  in  this  Memo- 
randum of  Understanding  will  be  resolved  by  mutual  coordination. 

c.  This  Memorandum  of  Understanding  is  subject  to  review  and 
revision  on  the  anniversary  date  of  the  Product  Manager  Charter  or  when 
major  program  changes  are  made  by  higher  headquarters. 

d.  This  Memorandum  of  Understanding  is  effective  on  date  of  signing 
by  the  Product  Manager,  CTS  and  Commandant,  USASIGS. 


G.  B.  HOWARD,  COL 
PRODUCT  MANAGER,  CTS 


DATE  4 December  1974 


CHARLES  R.  MYER,  MG 
COMMANDANT,  USASIGS 


DATE  13  January  1975 


3 Incl 


Project  ABACUS/CTS  Field  Office 


Directs  and  coordinates  the  design,  development,  test  and  evaluation  of 
Project  ABACUS  for  Army  training. 


Functions 


1.  Serves  as  program  director  and  principal  advisor  to  the  Product 
Manager  for  Project  ABACUS. 


2.  Serves  as  liaison  officer  and  principal  advisor  to  the  Commandant, 
USASIGS,  on  all  matters  relating  to  Project  ABACUS  design  and  develop 
ment  of  courses  and  operational  test  and  evaluation. 


3.  Monitors  USASIGS  participation  in  Project  ABACUS 


5.  Interfaces  with  the  DA  Steering  Advisory  Group  (SAG),  US  Army 
Computer  Systems  Support  and  Evaluation  Agency  (CSSEA),  US  Army 
Research  Institute  (ARI),  Human  Resources  Organization  (HumRRO)  and 
designated  contractors  and  consultants. 

6.  Augments  the  course  development  and  systems  programming/operation 
capability  of  the  Commandant,  USASIGS,  as  regards  Project  ABACUS. 


Proposed 

Project  ABACUS/CTS  Field  Office  Composition 


Position 

Grade 

MOS 

Br 

Req 

Auth 

Chief 

05 

04300 

SC 

1 

1 

Admin  NCO 

E7 

31G40 

SC 

1 

1 

Clerk  Typist 

03 

00322 

GS 

1 

1 

Instructional  Tech 

Team 

Chief 

13 

01710 

GS 

1 

1 

Ed  Spec 

12 

01710 

GS 

1 

1 

Ed  Spec 

11 

01710 

GS 

2 

2 

Crse  Writer 

E7 

32E40 

SC 

1 

1 

Crse  Writer 

E6 

32E40 

SC 

1 

1 

Crse  Writer 

E6 

31E40 

SC 

2 

2 

Crse  Writer 

E6 

31J40 

SC 

2 

2 

System  Prog  Team 

Chief 

03 

02402 

SC 

1 

1 

Sr  Prog  Sp 

E7 

74F20 

SC 

1 

1 

Prog  Sp 

E6 

74F20 

SC 

1 

1 

Eval  Team 

Chief 

12 

01710 

GS 

1 

1 

Statistician  Anal 

02 

SC 

2 

0 

TOTAL  19 


17 


CTS  Task  Group  Composition/ Representation 


2.  31J  Course 


a.  Computer  Programmers 


b.  Computer  operators 


c.  Educational  specialists 


b.  Evaluation  Br 


1.  Lib/Eearning  Center 


7.  Academic  Department  Representatives 


Operational  Test  and  Evaluation 
Organizational  Relationships 
USASIGS/CTS 


Augments  course  TM's  under  the  operational 
control  of  USASIGS  Task  Group  Leader. 


ANNEX  D 

PERSONNEL  ROSTER  - PROJECT  ABACUS 


Name 

Rank/ 

Grade 

Job  Title 

ROBBINS,  WILLIAM 

LTC 

Prog  Di r 

FOSTER,  ROBERT  G. 

LTC 

Prog  Dir 

KIMBERLIN,  DONALD  A. 

GS-13 

Chief,  Tech 
Appl  Div 

HARTMAN,  LARRY  K. 

CPT 

ADP  Off 

HAINES,  B.  E. 

GS-12 

Ed  Spec 

MUSSELWHITE,  HARRY  A 

GS-12 

Ed  Spec 

ALTMAN,  BRYAN  D. 

GS-1  1 

Ed  Spec 

LAMB,  JANET  M. 

GS-11 

Ed  Spec 

BROWN,  DONALD  L. 

SFC 

Instr  Prog 

BURNS,  WILLIAM  T. 

3SG 

Instr  Prog 

DIXON,  JOHN  W.  JR. 

SSG 

Instr  Prog 

HOOKER,  BERNARD  L.JR 

SSG 

1 nstr  Prog 

STOTTS,  JAMES  D. 

SSG 

Instr  Prog 

DUNCAN,  WILLIAM  L. 

SP5 

Programmer 

HUTSKO,  GARRETT  L. 

SP5 

Programmer 

MALCOLM,  VARA  G. 

SP5 

Programmer 

iMASHEY,  JOEL  A. 

SP4 

Programmer 

PEAK,  EDWIN  R.  1 1 1 

SP4 

Programmer 

ZARSKY,  DAVID  J. 

PV2 

Programmer 

JANTZEN,  RAILI  A. 

GS-3 

Clk-Typist 

D-' 
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ANNEX  F 


I 


INSTRUCTIONAL  PROGRAMMER  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM 


i 

1 


The  purpose  of  this  survey  is  to  obtain  your  assessment,  comments,  problems 
mu  ounterod,  and  suggestions  concerning  the  instructional  programmer's 
developmental  process.  The  results  of  this  survey  will  help  evaluate  areas 
where  developmental  problems  exist  and  serve  as  guidelines  for  future  CTS 
development  of  other  courses. 


DATE COURSE:  31E20  31J20  35L20 

PART  I:  Attitude  Toward  CTS  Model  Strategies  and  Course  Development 

Please  check  the  appropriate  space  to  indicate  your  opinion  of  the  following 
statements.  Other  comments  you  may  have  will  be  appreciated. 

1.  The  instructional  model  is  sensitive  to  the  abilities  and  needs  of  the 
individual  students: 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 


£ 

’ 

2.  The  instructional  model  is  flexible  enough  to  permit  a multimedia 
approach  to  instruction. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

■ 

Comments: 
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INSTRUCTIONAL  PROGRAMMER  SURVEY 

PART  I;  Attitude  Toward  CTS  Model  Strategies  and  Course  Development  (Cont) 

3.  The  instructional  model  is  not  suitable  for  both  off-line  management  of 
lessons  and  for  CAI  lessons. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 


4.  Recycling  of  students  under  computer  control  using  the  instructional  model 
and  strategies  should  be  effective. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 


5.  The  instructional  model  is  flexible  enough  to  permit  the  application  of 
diverse  instructional  strategies. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments : 
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INSTRUCTIONAL  PROGRAMMER  SURVEY 


6.  The  instructional  model  has  sufficient  latitude  to  permit  the  use  of 
existing  instructional  strategies. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 

7.  The  instructional  model  is  too  restrictive  to  enable  easy  modification  of 
the  existing  instructional  strategies. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 

8.  The  instructional  model  tends  to  emphasize  the  preparation  of  lesson 
material  for  computerized  presentation  rather  than  the  needs  of  the  student. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 


f 


F-3 


w 


INSTRUCTIONAL  PROGRAMMER  SURVEY 


PART  I:  Attitude  Toward  CTS  Model  Strategies  and  Course  Development  (Cont) 


9.  Since  no  instructional  strategy  has  attained  the  ideal  interaction  between 
each  student  and  the  subject  matter,  which  strategies  available  for  use  do  you 
think  need  the  most  improvement? 

Pretest  Preskill 

Posttest  Skill 

TAIS  Practice 

Lesson 

Please  give  suggestions  for  improvement. 


10.  It  is  easy  for  the  instructional  programmer  to  write  course  materials 
compatible  with  the  instructional  model  and  strategies. 

Strongly  Agree , Agree, Neutral,  Disagree,  Strongly  Disagree 


F-4 


Comments: 


INSTRUCTIONAL  PROGRAMMER  SURVEY 


PART  I:  Attitude  Toward  CTS  Model  Strategies  and  Course  Development  (Cont) 


11.  Courses  put  on  the  computer  should  be  prepared  by  instructors  who  have 
day  to  day  contact  with  students,  rather  than  a separate  group  of  instructional 
programmers. 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 


12.  Any  qualified  instructor  can  develop  course  material  for  a computerized 
training  system: 

Strongly  Agree,  Agree,  Neutral,  Disagree,  Strongly  Disagree 

Comments: 


13.  Give  the  percentage  (%)  of  total  course  development  time  (to  date)  that  has 
been  lost  because  a computer  terminal  was  unavailable. 


INSTRUCTIONAL,  PROGRAMMER  SURVEY 


PART  I:  Attitude  Toward  CTS  Model  Strategies  and  Course  Development  (Cont) 


14.  Show  break  out  of  lost  course  development  time  indicated  in  Question  13. 
Total  should  equal  100%  of  lost  time. 

% Author  load 

% Student  load 

% Computer  failure 

% Terminal  failure 

% Other  Input/Output  failure 

% Poor  response  time 

% Assembly  failure 

% Merge  failure 

% Lesson  code  lost 

% Parity  errors 

% Other  (identify): 

100  % Total 


Comments: 
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INSTRUCTIONAL  PROGRAMMER  SURVEY 

PART  II:  Authoring  Language 

Please  ( heck  the  appropriate  block  to 
• indicate  your  opinion  of  the  following 
statements.  Other  comments  you  may 
have  will  be  appreciated. 


15.  It  is  easy  to  create  text  displays 
using  CLASS  I language 


16.  It  is  difficult  to  create  graphics 
using  CLASS  I language. 


17.  It  is  difficult  to  code  for  impie 
mentation  of  strategies  using 
CLASS  I language. 


18.  Generally  on  line  authoring  is 
easy. 


19.  On  line  editing  of  displays  is 
difficult. 


20.  On  line  debugging  of  logic  is 
easy. 


21.  On  line  graphic  creation  is  more 
difficult  than  off  line  coding 
for  graphics. 


22.  CLASS  I language  meets  the  needs 
for  the  computer  training  system 
at  the  present  time. 


Comments 


b.  After  assignment: 

I 


24.  What  guidelines  or  suggestions  would  you  give  to  a newly  assigned 
instructional  programmer? 


25.  List  the  advantages  you  have  identified  in  using  the  one-man  concept 
(author,  audio-visual  specialist,  logic  coding)  in  developing  course  material. 
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INSTRUCTIONAL  PROGRAMMER  SURVEY 
PART  III:  General 


26.  List  the  disadvantages  you  have  encountered  in  employing  the  one-man 
concept  for  course  development. 


27.  What  problems  have  you  encountered  using  the  one-man  concept  of  course 
development  ? 


28.  If  you  were  in  a position  to  dictate  policy  for  future  CTS  development  in 
other  service  schools,  what  changes  would  you  make  to  the  existing  concept 
of  course  development? 
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STATUS:  Military  ( ) Civilian  ( ) 


Date  of  Birth 


TOTAL  MILITARY  SERVICE  (if  applicable) 


EDUCATION: 


CIVILIAN  SCHOOLS: 

Month  and  Year  Name  and  Location  of  School  Graduate  Degree 


SERVICE  SCHOOLS,  TECHNICAL  OR  SPECIALIZED  TRAINING  (Do  not  include 
those  less  than  five  days  duration): 

Length  of  Course  Course  Title  or  Area  of  Specialization 


How  long  have  you  been  working  in  an  Instructional  Programmer  position 
(months)  ? 


How  long  have  you  worked  as  an  Instructor  at  a Service  School  (years,  months)? 
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ANNEX  G 

STUDENT  ATTITUDE  QUESTIONNAIRE 
SELF-PACED  INSTRUCTION 

This  is  not  an  information  test.  Therefore,  it  has  no  right  or  wrong 
answers . Rather,  we  are  interested  in  your  candid  opinion  of  the  follow- 
ing statements . Your  complete  frankness  is  answering  these  questions 
will  be  greatly  appreciated.  Individual  responses  will  be  held  in  strictest 
confidence . 


Please  check  the  appropriate  block 
to  indicate  your  opinion  of  the 
following  statements . Explain 
your  selection  in  the  comments  if  j 
necessary. 


jsS 

Sy  vf  t 


tj  P/  oY 

, / "70/  «o7 


Comments 


1 . Overall  the  course  content 
holds  my  interest. 

2.  The  material  in  each  lesson 
is  organized  in  a way  that  I 
can  learn. 


3 .  The  objectives  of  the  course 
are  clear  and  I know  what 
is  expected. 


4.  Generally  the  lessons  are 
hard  to  understand. 


5.  The  lesson  material  re- 
quires you  to  think. 


6.  Generally  the  examinations 
cover  what  is  presented  in 
the  lessons. 


7.  Generally  the  lessons  are 
too  long. 


8.  I can  not  learn  what  I want 
to  'earn  in  this  kind  of  in- 
struction. 


y.  The  level  of  reading  skill 
required  in  the  lessons  is 
too  high. 


10.  I find  myself  harrying 

through  the  lessons  to  get 
it  over  with  rather  than 
trying  to  learn. 


11.  I answer  questions  wrong 
(pretest,  quiz,  etc.jinten- 
tionallv  to  get  more  mstruc- 


12.  Generally  the  lessons  seem 
to  be  planned  just  for  me. 

13.  I learn  the  ias~on  material 
vary  quic'dv  using  this 
method  o'  instruction. 


■ 


Please  check  the  appropriate  block 
to  indicate  your  opinion  of  the  / ^ 

following  statements . Explain  / ^ 
your  selection  in  the  comments  I gj 
if  necessary.  / #7 

©/  / / 
fit  j j 

Ji/i 

” 7 <S 

' , 4>/ 

j §l 

/ .<q/ 

1 y 

§/  Comments 

14.  I feel  that  electronics 

instruction  can  be  tailored 
to  suit  my  training  needs 
using  this  method  of  in- 
struction. 

• 

15.  1 feel  I can  retain  elec- 
tronics material  (forget 
less)  using  this  method  of 
instruction. 

1 

1 

1 

1 

1 

16.  I pay  more  attention  using 
this  method  of  instruction. 

1 

1 

l 

17.  1 waste  no  time  using  this 
method  of  instruction 

1 

1 

1 

1 

18 . I always  know  how  well  I 
am  doing  in  this  course . 

1 

1 

1 

1 

19 . I am  not  afraid  of  making 
mistakes  using  this 
method  of  instruction. 

1 

1 

1 

1 

20.  1 do  my  best  as  a result 
of  this  method  of  instruc- 
tion. 

1 

1 

1 

1 

21.  This  method  of  instruction 
is  a very  effective  method 
of  instruction. 

1 

1 

1 

1 

"22 . I feel  that  no  one  really 
cares  whether  I learn  or 
not  using  this  method  of 
instruction . 

1 

1 

1 

23.  The  instructors  can  an- 
swer my  questions. 

1 

1 

1 

24.  There  is  a good  working 
relationship  between  the 
instructors  and  myself. 

25.  I feel  that  I am  pushed  too 
quickly  through  the  lesson 
material. 

26.  An  instructor  is  readily 
available  for  assistance. 

- 

Please  check  the  appropriate  block 
to  indicate  your  opinion  cf  the 
following  statements . Explain 
your  selection  in  the  comments 
if  necessar 


A*  A? 

' ^ A i 


27.  I do  not  have  enough  inter- 
action with  other  students. 

28.  My  work  and  movements  are 
watched  too  closely  in  this 
method  of  instruction. 

\ 

6- 

29.  Background  noise  (voices  and 
personnel  movement)  is  dis- 
tracting. 

30.  The  lighting  iD  the  work  area 
is  adequate. 

l 

31.  There  is  enough  space  in  the 
working  areas  or  carrels  to 
work  comfortably. 

j 

32.  Working  in  the  working  areas 
or  carrels  over  a long  period 
of  time  becomes  tiring. 

■ 

\ 

l 

33.  Constant  movement  from  one 
instructional  media  to  another 
interferes  with  learning. 

There  are  too  many  devices 
(TV  cassettes,  slide  projec- 
tors , etc . ) to  operate  that  it 
detracts  from  the  instruction. 
There  is  too  mucn  operating 
noise  from  the  equipment. 


The  audio  is  clear  and  easily 
understood. 


There  is  too  much  material 
resented  on  the  slides. 


The  pictures  on  the  slides  are  i 
clear  and  do  not  cause  exces-  j 
sive  eyestrain. 


The  instructional  devices  are 
dilficu't  to  open  to 


(Specify) 


42.  What  do  you  like  best  about  this  method  of  instruction: 

a.  I can  go  at  my  own  speed. 

b.  It  presents  material  in  a clear  and  interesting  way. 

c . Iam  always  being  asked  questions . 

d.  I like  the  freedom  offered  by  a less  formal  environment. 

e.  I am  not  bothered  by  an  instructor  except  when  I need  him. 

f.  Others  (specify) 


43.  What  do  you  like  least  about  this  method  of  instruction: 

a.  I cannot  ask  questions. 

b.  It  is  too  much  work. 

c.  Have  to  learn  to  operate  too  much  instructional  equipment. 

d.  ft  is  too  impersonal. 

e.  ft  leaves  out  too  much  information  that  an  instructor  would  provide. 

f.  Could  not  get  assistance  when  I needed  it. 


g.  Others  (specify)_ 


44.  Which  lessons  or  part  of  the  course  caused  the  most  difficulty? 
(Be  as  specific  as  possible  . ) 


45.  What  instructional  devices  or  classroom  (working)  conditions  cause 
the  most  annoyance?  (Be  as  specific  as  possible.) 


Derach  :h*  last  pa^e  and  then  put  the  questionnaire  (first  six  pages) 
ir  : .e  nciosen  envelope. 
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- DETACH  - 


Background  Information 

To  insure  that  all  students  have  completed  the  questionnaire, 
you  are  asked  to  answer  the  following  items,  detach  from  the 
questionnaire,  and  return  to  the  instructor.  Put  the  questionnaire 
in  the  inclosed  envelope,  seal,  and  return  to  the  instructor  also. 

DATE:  

NAME: 

COURSE:  

Check  the  approximate  time  you  have  been  in  this  course: 
3 weeks;  entire  course. 


- DETACH  - 


ANNEX  H 


TRAINING  EFFECTIVENESS 
DATA  ITEMS  AND  COLLECTION  METHODOLOGY 


FORMATIVE  30  June  1975 

A.  Instructional  Process 


1.  What  are  the  advantages/disadvantages  of  the  one-man 
concept  (author,  audio-visual  specialist,  logic  coding,  entry  specialist 
validation  and  revision)  in  the  development  of  CTS  instructional  mate- 
rials? 


o Method : The  opinion  of  the  respective  course  managers 

is  important  in  the  assessment  of  the  one-man  concept  because  of  the 
impact  it  will  have  on  the  method  employed  in  the  preparation  of  CTS 
instructional  materials  in  The  future.  Data  for  this  item  will  be 
gathered  by  questionnaires  directed  to  the  instructional  programmers 
and  course  supervisory  personnel. 

2.  How  much  time  is  required  to  convert  the  training  mate- 
rials contained  in  the  operational  self-paced  course  annexes  to  CTS 
instructional  materials? 

o Method : This  item  will  focus  on  two  instructional 

alternatives  - (1)  annexes  which  include  a substantial  amount  rf  in- 
struction in  the  CAI  mode  (computer  serves  as  +he  instruction'll  medium); 
and,  (2)  annexes  primarily  employing  the  CMI  mode  (material  presented 
via  off-line  media  under  computer  management).  The  data  for  this  item 
will  be  gathered  from  CTS  time  logs  and  questionnaires  completed  by 
the  instructional  programmers. 

3.  To  what  extent  does  CAI  interface  with  other  media  (tele- 
vision, audio/visuals,  performance  guides,  tape  recorders)  employed 

in  the  courses?  How  can  it  be  optimized? 

o Method : Data  for  this  item  will  be  gathered  from 

questionnaires  addressed  to  instructors,  instructional  programmers, 
course  supervisors  and  members  of  the  staff  concerned  with  review  and 
approval  of  course  design. 

4.  To  what  extent  has  the  test  distractor  counts  and  col- 
lection of  unanticipated  responses  provided  feedback  necessary  for  re- 
vising tests  and  instructional  materials? 


o Method : The  data  for  this  Item  will  be  collected  by 

observation  and  item  question  sheets  directed  to  the  instructional 
programmers.  The  primarv  concern  will  be  whether  the  test  feedback 
will  highlight  any  deficiencies  in  the  instructional  materials  or  test 
questions  per  se. 

5.  Has  any  difficulty  been  encountered  in  fitting  the  pre- 
viously developed  self-paced  instructional  materials  into  the  CTS 
instructional  model? 

o Method : Observations  and  questionnaires  will  be  used 

to  develop  the  data  on  this  item.  The  major  thrust  will  be  to  deter- 
mine whether  the  instructional  programmers  experienced  any  reel  prob- 
lems in  adapting  the  existing  training  materials,  to  include  the  se- 
quencing of  Instruction,  to  the  CTS  instructional  model;  and  identify 
those  adjustments  or  modifications  that  were  necessary.  Input  to  this 
item  will  be  made  by  course  development  personnel. 

6.  What  is  the  optimum  on-line  time- length  per  session  for 
individual  students? 


5 
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o Method : The  data  for  this  item  will  be  collected  by 

observations  and  questionnaires  from  users.  The  student's  opinion  is 
essential;  as  the  sole  user  he  is  dependent  on  the  computer  for  his 
progress  through  the  course  and  should  have  the  most  valid  assessment 
of  the  optimum  on-line  time-length  per  session. 

B.  Support  Functions 

7.  What  special  problems,  if  any,  were  encountered  when  en- 
tering training  materials  on-line? 

o Method : Observations  and  item  question  sheet  will  be 

employed  to  obtain  this  data.  Instructional  programmers  will  be  the 
source  for  this  item. 

8.  Has  the  course  Program  of  Instruction  remained  stable 
during  the  transition  from  self-paced  to  CTS? 

o Method : Data  for  this  item  will  be  obtained  from  a 

review  of  systems  engineering  documents. 

9.  To  what  extent  does  the  system  enable  timely  modification, 
revision  and  evaluation  of  course  materials? 
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o Method : The  primary  objective  of  this  item  will  be  to 

determine  the  administrative  time  inherent  in  the  review  and  approval 
process  as  it  impacts  on  the  introduction  of  new  instructional  materi- 
al info  the  system.  This  data  will  be  gathered  by  observations  and 
questionnaires  directed  to  instructional  programmers,  course  super- 
visory personnel,  and  members  of  the  staff  involved  in  the  review  and 
approval  of  course  materials. 


10.  Was  it  necessary  to  assign  additional  instructional  prog- 
rammers to  course  development  in  order  to  prepare  back-up  training 
materials  to  be  utilized  during  computer  down  time?  Were  other  in- 
structor man-hours  devoted  to  this  task? 

o Method : Data  for  this  item  will  be  gathered  by  ques- 

tionnaires directed  to  course  managers,  instructional  programmers  and 
other  course  development  personnel  concerned.  The  primary  object  will 
be  to  pinpoint  any  personnel  resources  dedicated  exclusively  to  the 
preoaration  of  CTS  back-up  training  materials  per  se,  over  and  above 
those  needed  to  prepare  ongoing  self-paced  and  CTS  instructional  mate- 
rials. 


11.  What  is  the  average  development  time  (hours)  required  for 
one  hour  of  instruction  in  the  CAi  mode? 

o Method : The  CTS  Monthly  Time  Reports  will  provide  in- 

put for  this  item.  However,  the  data  gathered  may  be  somewhat  subjec- 
tive in  nature  as  it  may  be  necessary  to  include  "best  estimates"  of 
course  development  personnel  concerned.  Questionnai res  to  course 
managers  and  instructional  programmers  will  be  employed  in  this  in- 
stance. 

II  ADMINISTRATIVE 
A Reports 

1.  Do  recurring  student  activity  reports  provide  adequate 
and  timely  information  concerning  student  progress,  achievement,  and 
graduation  predictions? 

o Method : The  data  concerning  the  effectiveness  and 

utilization  of  the  weekly  student  activity  and  graduation  reports  will 
be  gathered  by  observations,  and  questionnaires  to  course  managers  and 
staff  personnel  concerned. 
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2.  Does  the  system  provide  the  respective  courses  with  the 
necessary  on-going  operational  reports? 

o Method:  Data,  with  regard  to  the  format,  quantity, 

usability,  and  content  of  the  recurring  operational  reports,  will  be 
obtained  by  observations  and  questionnaires  addressed  to  course  mana- 
gers and  supervisory  personnel. 

3.  Does  the  system  provide  real  time  access  to  student  data 
files,  both  on-line  and  printouts,  as  required  by  the  instructors  and 
course  managers? 

o Method : Observations  and  questionnaires  directed  to 

instructors  and  course  managers  will  be  utilized  to  provide  the  input 
for  this  item.  Information  will  be  obtained  as  to  the  ability  of  the 
system  to  support  the  counseling,  guidance,  and  faculty  board  require- 
ments and  concurrently  administer  student  progress  through  the  course. 

4.  What  is  the  impact  on  the  administrative  processing  of 
student  records  when  the  system  is  down  and  the  flow  of  recurring  re- 
ports is  halted  temporarily? 

o Method : Observations  and  questionnaires  directed  to 

system  operators  and  department  administrative  personnel  will  be  used 
to  gather  the  data  on  this  item.  The  information  obtained  will  estab- 
lish whether  or  not  the  student  data  can  be  retrieved  and,  if  not, 
whether  this  will  cause  any  delay  in  student  processing  or  graduation. 

5.  What  problems  were  encountered  in  updating  the  student 
activity  reports  following  computer  down  time? 

o Method : The  data  for  this  item  will  be  collected  by 

observation  and  questionnaires  addressed  to  course  managers  and  de- 
partment administrative  personnel.  In  addition  to  any  unanticipated 
problems  that  may  surface,  any  requirement  to  maintain  a dual  set  of 
manually  maintained  student  records  will  be  specifically  addressed. 

B.  Resource  Allocation 


6.  Did  the  implementation  of  CTS  cause  any  changes  in  the 
accounting  of  student  time  in  the  course? 

o Method : Data  for  this  item  will  be  obtained  from  on- 

line programs,  instructor  logging  and  an  item  question  sheet  addressed 
to  the  classroom  instructors.  The  special  area  of  interest  will  be 
accounting  for  non-academic,  as  well  as,  academic  pursuits. 
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7.  Has  the  system  been  effective  in  maintaining  accountabi- 
lity of  students  within  the  respective  tasks/annexes? 

o Method : Observation,  and  item  question  sheets  directed 

to  primary  instructors  and  section  chiefs,  will  be  employed  to  gather 
the  data  on  this  item. 

8.  Has  the  system  been  effective  in  directing  the  student 
progress  through  the  course  via  the  several  learning  alternatives? 

o Method:  Data  for  this  item  will  be  gathered  by  obser- 

vations and  questionnaires  directed  to  the  instructional  programmers 
and  course  managers. 

9.  Has  the  system  been  successful  in  routing  students  through 
the  respective  courses  according  to  standard  flow  or  prior  accomplish- 
ments, or  to  alternate  tasks  and  positions  on  space  available  basis? 

o Method : (Same  as  II.B.8) 

10.  Does  the  administrative  burden  of  collection  outweigh  the 
value  of  accounting  for  student  time  within  the  various  tasks? 

o Method : Observations  and  a questionnaire  addressed  to 

the  instructors  will  be  used  to  collect  this  information.  This  data 
will  provide  an  insight  into  the  relative  value  of  detailed  accounting 
of  students'  time  in  each  learning  element  and  will  be  geared  to  future 
app  I ications. 

11.  What  changes,  if  any,  are  recommended  to  the  TRADOC  School 
Staffing  Guide  to  accomodate  the  unique  role  of  the  instructional  prog- 
rammer? 

o Method : Data  for  this  item  will  be  obtained  by  obser- 

vations and  questionnaires  directed  to  course  managers  and  to  management 
and  operations  personnel  at  department  level. 

C.  instructional  Process 

12.  What  is  the  optimum  student  position/terminal  relationship? 

o Method : The  data  for  this  ’tern  will  be  gathered  by  ob- 

servations and  item  question  sheets  directed  to  students,  instructors 
and  course  managers.  The  information  obtained  will  provide  an  insight 
into  the  number  of  terminals  required  in  relationship  to  the  number  of 
student  positions  which  will  insure  maximum  utilization  of  the  system 
and,  at  the  same  time,  prevent  queuing. 
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13.  What  changes  in  the  "reward  for  early  completion"  have 
occurred  that  may  have  impacted  on  student  motivation? 

o Method : The  data  for  this  item  will  be  found  in 

changes  to  promotion  policies,  assignment  instructions,  or  other  re- 
gulations that  may  impinge  on  the  student's  desire  for  early  comple- 
tion (e.g.,  reduction  in  number  of  promotions  for  high  academic 
stand ing) . 

14.  Is  there  an  adequate  back-up  capability  to  provide  in- 
struction during  computer  down  time? 

o Method : Data  for  this  item  will  be  obtained  by  an 

item  question  sheet  directed  to  instructional  programmers  and  section 
supervisors  and  it  will,  specifically,  address  the  production  of  hard 
copy  that  can  be  used  for  this  purpose. 

15.  Has  the  implementation  of  CTS  increased  the  interaction 
time  between  student  and  instructor? 

o Method : A major  problem  in  self-paced  courses  has  been 

the  non-availability  of  the  instructor  tc  the  student  when  the  occasion 
arises.  Data  for  this  item  will  be  gathered  by  an  item  questionnaire 
directed  to  instructors  and  course  supervisory  personnel. 

D.  Support  Functions 

16.  What  administrative  and  personnel  resources  were  required 
to  establish  training  programs  for  CTS  Instructors  and  Workshops  for 
Instructional  Programmers  and  IPES  personnel? 

o Method : Data  for  this  item  will  be  obtained  from 

USASIGS  Faculty  Development  and  Data  Systems  Branches. 

17.  What  special  qualifications  are  required  by  the  Instruc- 
tional Program  Entry  Specialists  (IPES)? 

o Method : Observations  and  questionnaires  addressed  to 

instructional  programmers  and  administrative  supervisors  will  be  uti- 
lized to  provide  data  input  for  this  item. 

18.  Can  clerk  typist/key  punch  operators  double  as  IPES  ef- 
fective I y? 


o Method : (Same  as  D.17  above) 


19.  Is  the  system  capable  of  maintaining  uninterrupted  in- 
struction and  simultaneously  process  student  data  in  real  time? 

o Method : Data  for  this  item  will  be  collected  by 

observations  and  an  item  question  sheet  addressed  to  systems  opera- 
tional personnel. 

20.  What  is  the  average  in-house  (USASIGS)  cost  per  student 
week  for  the  self-paced  courses  of  instruction  selected  for  the  field 
test?  For  the  same  courses  after  instituting  CTS? 

o Method : Data  for  this  item  will  be  obtained  from 

Management  and  Budget,  USASIGS. 

21.  What  unanticipated  side  effects  or  by-products  can  be 
attibuted  to  the  implementation  of  the  computerized  training  system? 

o Method : Observations,  questionnaires  and  spot  reports 

will  be  used  to  gather  data  for  this  item.  Input  will  be  gathered 
from  USASIGS  and  Project  ABACUS  personnel  concerned  with  the  CTS  field 
test. 

Ill  SUMMATIVE 

A.  Within  and  Across  Courses 


1.  Is  there  any  significant  difference  in  task/annex  train- 
ing time  before  and  after  conversion  to  CTS? 

o Method : Wherever  feasible  at  the  task/annex  level, 

the  student  Progression  Index  (PI),  self-paced  vs  CTS,  will  be  examined 
to  determine  whether  the  mean  performances  of  the  two  groups  are  signi- 
ficantly different.  The  t test  will  be  used  to  determine  whether  the 
difference  between  the  two  means  is  significant.  It  should  be  noted 
that  the  integrity  of  all  tasks  will  not  be  maintained  throughout  the 
field  test  period  because  of  revisions  to  the  program  of  instruction 
and  translation  of  self-paced  instructional  materials  to  CTS.  Data 
for  this  item  will  be  gathered  from  the  daily/weekly  student  activity 
reports. 


2.  What  differences  have  been  discerned  in  the  failure  and 
attrition  rates  in  the  respective  courses  before  and  after  conversion 
to  CTS? 

o Method : The  data  for  this  item  will  be  obtained  from 

the  Monthly  Course/Test  Report  and  the  Quarterly  Review  and  Analysis 
(USASIGS)  recorded  in  matrix  form. 
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3.  Has  the  percentage  of  no-goes  (performance  test  failures) 
changed  substantially  since  the  Implementation  of  CTS? 

o Method : The  data  for  this  Item  will  be  obtained  from 

daily/weekly  student  activity  reports  using  on-line  programs  where 
feasible.  Collection  and  analysis  will  be  limited  to  those  tasks/ 
annexes  wherein  the  integrity  of  the  task  has  been  maintained  for  both 
CTS  and  self-paced  instruction. 

4.  Does  the  system  have  the  capacity  to  provide  instruc- 
tional and  administrative  support  concurrently  for  (1)  single  shift, 

(2)  double  shift  operation? 

o Method:  Data  for  this  item  will  be  collected  by  ob- 

servations and  questionnaires  addressed  to  instructional  programmers, 
system  programmers,  and  system  operators. 

5.  Does  second  shift  operation  present  the  same  administra- 
tive and  instructor  requirements? 

o Method : Data  for  this  item  will  be  gathered  by  obser- 

vations and  questionnaires  addressed  to  course  managers  and  supervisors. 

6.  Has  the  system  been  effective  in  managing  the  student's 
off-line  activities  (CMI)? 

o Method : Since  the  student  will  spend  only  a limited 

time  Interacting  with  the  terminal  (CAI),  it  is  necessary  to  know  how 
he  responds  to  computer-directed  activities.  Input  for  this  item  will 
be  obtained  from  students  and  instructors.  Observations  and  question- 
naires will  be  used  to  collect  this  data. 

7.  What  problems  have  been  encountered  in  managing  the 
student  flow? 


o Method : The  prime  target  for  this  item  will  be  to 

Identify  and  isolate  any  unique  or  unusual  problems  that  have  surfaced 
in  the  area  of  managing  or  directing  the  student  progress  through  the 
respective  courses.  This  data  will  be  collected  by  questionnai res  and 
will  impact  on  both  on-line  and  off-line  activities. 

8.  Was  it  necessary  to  make  any  changes  or  revisions  to  the 
self-paced  instructional  materials  before  they  could  be  incorporated 
Into  the  system? 
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o Method : Instructional  programmers  will  provide  the 


input  for  this  item.  The  primary  concern  will  be  to  ascertain  the  com- 
patibility of  the  on-going  course  materials  with  the  computer  mode  and, 
determine  how  much  rewrite  or  revision  was  necessary  to  convert  these 
materials  to  CTS. 


CTS? 


Are  graduation  predictions  more  timely  since  conversion  to 


o Method : Data  for  this  item  will  be  obtained  by 

spection  of  the  student  activity  reports  and  depicted  by  matrix 


i n- 
and 


histogram  as  appropriate. 


10.  Which  student  aptitude  (ACB)  area  appears  to  have  the  most 
likelihood  of  predicting  student  success? 


o Method : Data  for  this  item  will  be  gathered  from  the 


student  evaluation  rosters  and  daily/weekly  student  activity  reports. 
Since  only  one  aptitude  area  can  be  cited  as  a prerequisite  for  atten- 
dance to  a TRADOC  School  course  and  is  used  exclusively  by  school  per- 
sonnel in  managing  the  student  through  a course,  it  is  essential  that 
the  best  predictor  of  achievement  be  determined.  The  single  predictor 
regression  scheme  will  be  used  in  making  this  determination. 


11.  Students,  instructional  programmers,  instructors,  and 
members  of  the  staff  and  faculty,  USASIGS,  will  be  surveyed  to  deter- 
mine their  attitudes  and  opinions  of  the  existing  self-paced  courses 
and  of  the  computerized  training  system  (CTS)  after  its  implementation. 
Topical  areas,  in  general,  are  listed  below: 


Students : 


(1)  Course  content 

(2)  Instructional  methods 

(3)  Instructiona I media 

(4)  Individualization  of  instruction 

(5)  Acceptance  of  sel f-pacing/CTS 

(6)  Training  conditions 
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I 

: 


I 


b.  Instructor: 

(1)  All  of  pa~agraph  9a  above 

(2)  Instructor  role  (problems) 

(3)  Instructor/student  relationship 

c.  Instructional  Programmer: 

(1)  'Course  strategies 

(2)  Course  development 

(3)  Authoring  language 

(4)  Training 

(5)  One-man  concept 

d.  Staff  and  Faculty: 

(1)  Acceptance  se I f-paced/CTS 

(2)  Instructional  problems/innovations 

(3)  Advantages/disadvantages 

e.  Other  related  items,  as  appropriate,  may  also  be 
addressed  in  these  surveys. 

o Method : Surveys/quest ionnai res  will  be  used  to  collect 

this  data.  Matrices  will  be  developed  to  record  and  quantify  unstruc- 
tured responses. 

B.  Specified  Task  Analysis  (One  task  selected  in  each  course  will 
be  subjected  to  comparative  analysis) 

1.  Is  there  any  significant  difference  in  the  student  progres- 
sion index  between  self-paced  and  CTS  for  the  specified  task? 

o Method : Student  Pis  will  be  examined  to  determine 

whether  the  mean  performance  of  the  two  groups  are  significantly  differ- 
ent. The  t test  will  be  used  to  determine  the  difference. 
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2.  Does  CTS  require  as  much  instructor  time  for  student 
record  keeping  as  seif-paced? 

o Method : Data  for  this  item  will  be  collected  by  in- 

dividual instructors  by  logging  and  input  by  the  respective  courses. 

Applicable  only  to  student  records  per  se. 

3.  Has  the  computer  been  more  effective  in  monitoring  stu- 
dent progress  than  the  present  i nstructor/summary  training  record? 

o Method : Input  for  this  item  will  be  primarily  from 

the  course  instructors  and  supervisors.  The  student  control  document 
in  self-paced  is  the  Student/Summary  Training  Record.  Terminal  com- 
mands and  sign-off/return-to  messages  will  track  the  student  through 
CTS.  Data  will  be  collected  by  observations  and  item  questionnaires. 

4.  What  unique  problems  were  encountered  in  preparing  CTS 
instructional  materials  not  identified  when  preparing  self-paced  mate- 
rials? 

o Method : Observations  and  questionnaires  will  be  used  ] 

to  collect  this  data  from  instructional  programmers  and  course  mana- 
gers. The  review  and  approval  process  will  be  addressed  within  the 
scope  of  this  item. 

5.  Have  randomly  generated  performance  test  items  resulted 
in  more  effective  student  testing?  Time  saving  for  the  test  monitor? 

An  increase  in  test  failures? 

o Method : This  data  will  be  collected  by  observations 

and  questionnaires  directed  to  the  course  managers,  test  monitors  and 
primary  instructors. 

6.  What  advantages  have  been  derived  by  having  the  system 
record  successful  completion  of  critical  actions  inherent  in  task  per- 
formance tests? 

o Method:  Current  practice  dictates  that  the  test  ad- 

ministrator record  a "go"  or  "no-go"  for  the  entire  test.  Feedback  on 
critical  action  performance  is  net  formalized.  This  data  should  pro- 
vide the  course  development  personnel  with  another  means  of  validating 
instruction.  Observations  and  questionnaires  to  instructors  and  in- 
structional programmers  will  be  used  to  collect  this  data. 

7.  Has  the  implementation  of  CTS  reduced  queuing  time  at 
student  positions? 

o Method:  Data  for  this  item  will  be  obtained  by  obser- 

vations and  questionnaires  to  instructors. 
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ANNEX  I 


CONFERENCES  AND  PRESENTATIONS 


A.  Conferences: 


6-7  Aug  74 


12-16  Aug  74 


11-13  Sep  74 


4-6  Dec  74 


21  Apr  75 


17-18  Jun  75 


Policy  and  Procedures  Subgroup  Meeting  on  Development 

of  Proposed  TRADOC  Regulation  18 , Applications 

of  Computers  to  Training  and  Simulation,  Fort  Monroe, 
Virginia 

Association  for  the  Development  of  Computer  Based 
Instructional  Systems  (ADCIS),  Western  Washington 
College,  Bellingham,  Washington 

Author  Language  Conference,  US  Army  Research  Institute 
(ARI),  Rosslyn,  Virginia 

Consultant  Site  Visit  and  DA  SAG  Meeting,  US  Army 
Signal  School  (USAS1GS),  Fort  Gordon,  Georgia 

• 

Training  Resources  Application  information  Data 
Exchange  (TRAIDEX)  Meeting,  Defense  Advanced  Research 
Projects  Agency  (ARPA),  Arlington,  Virginia 

National  Security  Industrial  Association  (NSIA) 
Conference  on  Application  of  Advanced  Training  Tech- 
nology, US  Army  Signal  School,  Fort  Gordon,  Georgia 


B.  Presentations: 

17  Dec  74  COL  G.B.  Howard  - Presentation  on  CTS  - Institute  of 

Defense  Analysis,  Blue  Ribbon  Panel,  Institute  of 
Defense  Analysis,  Washington,  DC 
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ANNEX  J 


COMPUTERIZED  TRAINING  SYSTEMS  DIRECTORATE 
US  ARMY  TRAINING  SUPPORT  ACTIVITY 
FORI  EUSTIS,  VIRGINIA  23604 

PUBLICATIONS 


1 


TITLE 

CTS-TR  tt 

DDC  a 

A Feasibility  Study  of  Computer  Assisted 
Instruction  in  US  Army  Basic  Electronics 
Training.  Feb  68 

(None) 

AD  745 

402 

The  Implementation  of  Computer  Assisted 
Instruction  in  US  Army  Basic  Electronics 
Training.  Sep  69 

69-1 

AD  704 

339 

Audio  Utilization  Conventions  and  Tech- 
niques for  Computer  Assisted  Instruction. 
Mar  70 

70-1 

AD  704 

338 

An  Automated  Student  Registration  Pro- 
cedure (REGIS).  Jun  70 

70-2 

AD  710 

983 

A MACRO  System  for  Computer  Assisted 
Instruction.  May  70 

70-3 

(None) 

Application  of  Computers  to  Training. 
Apr  71 

71-1 

AD  749 

468 

An  Instructional  Model  for  Computer 
Assisted  Instruction.  May  71 

71-2 

AD  745 

409 

Instructional  Programming  Guide  for 
Computer  Assisted  Instruction.  Jul  71 

71-3 

AD  749 

469 

Task  Group  Report:  CAI  Volumes  1 & II. 

Apr  72 

(None) 

(None) 

A Summative  Evaluation  of  Computer 
Assisted  Instruction  in  US  Army  Basic 
Electronics  Training.  May  72 

72-1 

AD  749 

470 

Vacuum  Tube/Solid  State  Circuit  Survey. 
Mar  73 

73-1 

AD  759 

129 

CLASS  1 Language:  Document  A (Specif. 

No:  S-125-72) . Apr  73 

(None) 

(None) 

J-1 


TITLE 

CTS-TR  § 

DDC  # 

Record  Formats:  Booklet  A (Specif. 

No:  S- 125-72) . Apr  73 

(None) 

(None) 

Concept  Plan:  Booklet  B (Specif. 

No:  S-125-72) . Apr  73 

(None) 

( None ) 

Estimated  System  Use  Factors:  Document  C. 

(Specif.  No:  S-125-72).  Apr  73 

(None) 

(None) 

A Preliminary  Instructional  Model  for  a 
Computerized  Training  System.  Jul  73 

73-2 

AD  762 

180 

An  On-Line  Electronics  Graphics  Symbol 
Set  for  the  PLATO  IV  System.  Oct  73 

73-3 

AD  776 

364 

One  Year  Status  Report  Computerized 
Training  System:  Project  ABACUS.  Aug  73 

73-4 

AD  777 

767 

Preliminary  Evaluation  Plan  for  US  Army 
Computerized  Training  System.  Jan  74 

74-1 

AD  777 

783 

PLATO  IV  First  Year  Report  Computerized 
Training  System.  Apr  74 

74-2 

(None) 

Survey  of  Computer  Applications  in  Army 
Training.  Aug  74 

74-3 

AD  787 

429 

Second  Year  Status  Report  Computerized 
Training  Systems  Project  ABACUS.  Aug  74 

74-4 

(None) 

Effective  Writing  for  a Computerized 
Training  System.  Jan  75 

75-1 

(None) 

Computer  Applications  in  Army  Training 
Present  Status  and  Planned  Activity. 
Apr  75 

(None) 

(None) 

Instructional  Effectiveness  of  the 
PLATO  IV  Plasma  Terminal.  May  75 

75-2 

(None) 

The  Future  of  the  Computer  in  Army 
Training.  May  75 

75-3 

(None) 
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